

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>术语 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="MDS 缓存配置" href="../cache-configuration/" />
    <link rel="prev" title="部署元数据服务器" href="../add-remove-mds/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../">Ceph 文件系统</a></li>
      <li class="breadcrumb-item active">术语</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/cephfs/standby.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Ceph 文件系统</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../#cephfs">CephFS 入门</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../#id4">管理</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../createfs/"> 创建 CephFS 文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../administration/"> 管理命令</a></li>
<li class="toctree-l3"><a class="reference internal" href="../multifs/"> 创建多个文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-remove-mds/"> 配备、增加、删除 MDS</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">术语</a></li>
<li class="toctree-l3"><a class="reference internal" href="#mds">MDS 守护进程的引用</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id3">故障切换的管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="#standby-replay">热备（ standby-replay ）的配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="#mds-join-fs">配置 MDS 与文件系统的亲和性</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cache-configuration/"> MDS 缓存配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mds-config-ref/"> MDS 配置选项</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../man/8/ceph-mds/"> ceph-mds 手册页</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nfs/"> 通过 NFS 导出</a></li>
<li class="toctree-l3"><a class="reference internal" href="../app-best-practices/"> 应用最佳实践</a></li>
<li class="toctree-l3"><a class="reference internal" href="../fs-volumes/"> FS 卷和子卷</a></li>
<li class="toctree-l3"><a class="reference internal" href="../quota/"> CephFS 配额管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-messages/"> 健康消息</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/">升级 MDS 集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/#firefly-jewel">升级比 Firefly 老的文件系统，需过 Jewel 这个槛</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cephfs-top/"> CephFS Top 工具</a></li>
<li class="toctree-l3"><a class="reference internal" href="../snap-schedule/"> 定时快照</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cephfs-mirroring/"> CephFS 快照镜像</a></li>
<li class="toctree-l3"><a class="reference internal" href="../purge-queue/"> 清理队列</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../#id5">挂载 CephFS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id6">CephFS 内幕</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id7">故障排除和灾难恢复</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id9">更多细节</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="mds-standby">
<span id="id1"></span><h1>术语<a class="headerlink" href="#mds-standby" title="Permalink to this heading"></a></h1>
<p>一个 Ceph 集群内可以没有、或者有多个 CephFS <em>文件系统</em> 。每一个 CephFS 文件系统都有一个人类可读的名字（用 <code class="docutils literal notranslate"><span class="pre">fs</span> <span class="pre">new</span></code> 创建时设置的）、和一个整数 ID ，这个 ID 称为文件系统集群 ID 或 <em>FSCID</em> 。</p>
<p>每个 CephFS 都有几个 <em>rank</em> ，编号从 0 起。默认情况下，每个文件系统有一个 rank 。一个 rank 可看作是一个元数据分片。
rank 的管理在 <a class="reference internal" href="../multimds/"><span class="doc">多活 MDS 守护进程的配置</span></a> 里详述。</p>
<p>CephFS 的每个 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 进程刚启动时都没有 rank ，它可以由集群的监视器们分配。一个守护进程一次只能占据一个 rank ，只有在 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程停止的时候才会释放 rank 。</p>
<p>如果某个 rank 没有依附任何一个守护进程，那这个 rank 就*失效了（ <code class="docutils literal notranslate"><span class="pre">failed</span></code> ）*。分配给守护进程的 rank 才被当作 <em>正常的（ ``up`` ）</em> 。</p>
<p>首次配置 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程时，管理员需分配固定的<em>名字</em>，一般都会用守护进程所在的主机名作为其守护进程名字。</p>
<p>把一个 MDS 的 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 配置选项设置为指定文件系统的名字，
可以将 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程分配给这个文件系统。</p>
<p><code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程每次启动时，还会被分配一个整数 <code class="docutils literal notranslate"><span class="pre">GID</span></code> ，它对于当前这个守护进程的进程生命周期来说它是唯一的。换句话说，当一个 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程重启后，它会作为新进程、而且会被分配一个 <em>新的</em> 、不同于之前进程的 <code class="docutils literal notranslate"><span class="pre">GID</span></code> 。</p>
<div class="admonition-todo admonition" id="id2">
<p class="admonition-title">Todo</p>
<p>译者注： rank 翻译为“席位”、“座席”？它们共同处理元数据，
且动态分配，类似客服中心的座席。</p>
</div>
</section>
<section id="mds">
<h1>MDS 守护进程的引用<a class="headerlink" href="#mds" title="Permalink to this heading"></a></h1>
<p>与 <code class="docutils literal notranslate"><span class="pre">ceph-mds</span></code> 守护进程（MDS）相关的大多数管理命令都接受一个灵活的参数格式，可以指定 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 、 <code class="docutils literal notranslate"><span class="pre">GID</span></code> 或 <code class="docutils literal notranslate"><span class="pre">name</span></code> 。</p>
<p>使用 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 时，它前面可以加文件系统的 <code class="docutils literal notranslate"><span class="pre">name</span></code> 或 <code class="docutils literal notranslate"><span class="pre">GID</span></code> ，也可以不加。如果某个守护进程是灾备的（即当前还没给它分配 <code class="docutils literal notranslate"><span class="pre">rank</span></code> ），那就只能通过它的 <code class="docutils literal notranslate"><span class="pre">GID</span></code> 或 <code class="docutils literal notranslate"><span class="pre">name</span></code> 引用。</p>
<p>例如，假设我们有一个 MDS 守护进程，名为 myhost ，其 <code class="docutils literal notranslate"><span class="pre">GID</span></code> 为 5446 ，
分配的 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 是 0 ，它位于名为 myfs 的文件系统内，
文件系统的 <code class="docutils literal notranslate"><span class="pre">FSCID</span></code> 是 3 ，那么，下面的几种形式都适用于 <code class="docutils literal notranslate"><span class="pre">fail</span></code> 命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">mds</span> <span class="n">fail</span> <span class="mi">5446</span>     <span class="c1"># GID</span>
<span class="n">ceph</span> <span class="n">mds</span> <span class="n">fail</span> <span class="n">myhost</span>   <span class="c1"># Daemon name</span>
<span class="n">ceph</span> <span class="n">mds</span> <span class="n">fail</span> <span class="mi">0</span>        <span class="c1"># Unqualified rank</span>
<span class="n">ceph</span> <span class="n">mds</span> <span class="n">fail</span> <span class="mi">3</span><span class="p">:</span><span class="mi">0</span>      <span class="c1"># FSCID and rank</span>
<span class="n">ceph</span> <span class="n">mds</span> <span class="n">fail</span> <span class="n">myfs</span><span class="p">:</span><span class="mi">0</span>   <span class="c1"># Filesystem name and rank</span>
</pre></div>
</div>
</section>
<section id="id3">
<h1>故障切换的管理<a class="headerlink" href="#id3" title="Permalink to this heading"></a></h1>
<p>如果一个 MDS 守护进程不再与监视器通讯，监视器把它标记为 <em>laggy</em>
（滞后）状态前会等待 <code class="docutils literal notranslate"><span class="pre">mds_beacon_grace</span></code> 秒（默认是 15 秒）。如果有可用灾备，监视器会立即替换处于 laggy 状态的守护进程。</p>
<p>为安全起见，每个文件系统都要指定一些灾备守护进程，灾备数量包括处于 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 热备状态、等着接替失效 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 的。记着， <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 热备守护进程不会被重分配去接管另一个 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 、或者另一个 CephFS 文件系统内的失效守护进程）。不在 <code class="docutils literal notranslate"><span class="pre">replay</span></code> 重放状态的灾备守护进程们会被所有文件系统算作自己的灾备。每个文件系统都可以单独配置期望的灾备守护进程数量，用这个：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">fs</span> <span class="nb">set</span> <span class="o">&lt;</span><span class="n">fs</span> <span class="n">name</span><span class="o">&gt;</span> <span class="n">standby_count_wanted</span> <span class="o">&lt;</span><span class="n">count</span><span class="o">&gt;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">count</span></code> 设置为 0 表示禁用健康检查。</p>
</section>
<section id="standby-replay">
<span id="mds-standby-replay"></span><h1>热备（ standby-replay ）的配置<a class="headerlink" href="#standby-replay" title="Permalink to this heading"></a></h1>
<p>每个 CephFS 文件系统都能做配置，增加几个热备（ <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> ）守护进程。
这些备用守护进程会跟踪活动 MDS 的元数据日志，
以便在活跃 MDS 不可用时减少故障切换时间。
每个活跃 MDS 只能有一个 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 守护进程跟着它。</p>
<p>在文件系统上配置 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 的方法如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">fs</span> <span class="nb">set</span> <span class="o">&lt;</span><span class="n">fs</span> <span class="n">name</span><span class="o">&gt;</span> <span class="n">allow_standby_replay</span> <span class="o">&lt;</span><span class="nb">bool</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>设置以后，监视器将分配可用的备用守护进程，让它们跟随此文件系统中的活跃 MDS 。</p>
<p>一旦 MDS 进入 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 状态，它就只能用作跟随着的 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 的备机。
如果另一个 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 发生故障，即使没有其他备机，也不会用这个
<code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 守护进程作替代品。因此，如果启用 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> ，
建议<em>每个</em>活跃 MDS 都应该配备一个 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 守护进程。</p>
</section>
<section id="mds-join-fs">
<span id="id4"></span><h1>配置 MDS 与文件系统的亲和性<a class="headerlink" href="#mds-join-fs" title="Permalink to this heading"></a></h1>
<p>您可能会选择将 MDS 焊死到指定的文件系统上。或者，让 MDS 在较好的硬件上运行，
而不是在普通的或还有空余资源的系统上运行。要配置出这种优先级，
CephFS 为 MDS 提供了一个名为 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 的配置选项，
用于实现这种亲和性。</p>
<p>当 MDS 守护进程做故障切换时，集群的监视器会优先选择 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 值等于故障 <code class="docutils literal notranslate"><span class="pre">rank</span></code> 对应文件系统名字 <code class="docutils literal notranslate"><span class="pre">name</span></code> 的备用守护进程。
如果不存在 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 等于文件系统名的备机，
它将选择一个不合格的备用系统（没有设置 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> ）来替代。
在万不得已的情况下，系统会选择另一个文件系统的备机，
但这种行为可以禁用：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">fs</span> <span class="nb">set</span> <span class="o">&lt;</span><span class="n">fs</span> <span class="n">name</span><span class="o">&gt;</span> <span class="n">refuse_standby_for_another_fs</span> <span class="n">true</span>
</pre></div>
</div>
<p>注意，配置 MDS 的文件系统亲和性不会改变备机选择行为，
总是先选择 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 守护进程、再选其他备机。</p>
<p>此外，即使在稳定状态下，监视器也会定期检查 CephFS 文件系统，
以检查是否有亲和性更强的备机来替换亲和性较低的 MDS 。
这个过程也适用于 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> 守护进程：
如果常规备机的亲和性比 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> MDS 更强，
它就会取代 <code class="docutils literal notranslate"><span class="pre">standby-replay</span></code> MDS 。</p>
<p>例如，这是个稳定、健康的文件系统：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs dump
dumped fsmap epoch 399
...
Filesystem &#39;cephfs&#39; (27)
...
e399
max_mds 1
in      0
up      {0=20384}
failed
damaged
stopped
...
[mds.a{0:20384} state up:active seq 239 addr [v2:127.0.0.1:6854/966242805,v1:127.0.0.1:6855/966242805]]

Standby daemons:

[mds.b{-1:10420} state up:standby seq 2 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]]
</pre></div>
</div>
<p>给备机配置 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> ，展现自己的偏好：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph config set mds.b mds_join_fs cephfs
</pre></div>
</div>
<p>发生故障，自动切换后：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs dump
dumped fsmap epoch 405
e405
...
Filesystem &#39;cephfs&#39; (27)
...
max_mds 1
in      0
up      {0=10420}
failed
damaged
stopped
...
[mds.b{0:10420} state up:active seq 274 join_fscid=27 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]]

Standby daemons:

[mds.a{-1:10720} state up:standby seq 2 addr [v2:127.0.0.1:6854/1340357658,v1:127.0.0.1:6855/1340357658]]
</pre></div>
</div>
<p>注意上例中 <code class="docutils literal notranslate"><span class="pre">mds.b</span></code> 现在是 <code class="docutils literal notranslate"><span class="pre">join_fscid=27</span></code> 。
在此输出中， <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 中的文件系统名称已经改成了文件系统标识符 (27)。
如果以相同的名字重新创建文件系统，
备机将按照预期跟随新文件系统。</p>
<p>最后，如果文件系统降级或副本数降低，
故障转移不会影响 <code class="docutils literal notranslate"><span class="pre">mds_join_fs</span></code> 的施行。</p>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../add-remove-mds/" class="btn btn-neutral float-left" title="部署元数据服务器" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../cache-configuration/" class="btn btn-neutral float-right" title="MDS 缓存配置" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>